Contractive Nonlinear Neural Networks: Stability and Spatial Frequencies

We consider models of the form J.d: = -x + p + WF(x) , where x = x(t) is a vector whose entries represent the electrical activities in the units of a neural network. W is a matrix of synaptic weights, F is a nonlinear function, and p is a vector (constant over time) of inputs to the units. If the map WF(x) is a contraction, then the system has a unique equilibrium which is globally asymptotically stable; consequently the network's steady-state response to any constant input is independent of the initial state of the network. We consider also some relatively mild restrictions on Wand F(x), involving the eigenvalues of Wand the derivative of F, that are sufficient to ensure that WF(x) is a contraction. We show that in the case of spatially-homogeneous synaptic weights, the eigenvalues of Ware simply related to the Fourier transform of the connection pattern. This relation makes it possible, given cortical activity patterns as measured by autoradiographic labeling, to construct a pattern of synaptic weights which produces steady state patterns showing similar frequency characteristics. Finally we bound the norm of the difference between the equilibrium of the model and that of the simpler linear model Jd: = -x + p + Wx; this latter equilibrium can be computed simply from p in the homogeneous case using Fourier transforms. This paper represents research supported by NIDR Program Grant DE-07509 (B. L. Whitsel, Principal Investigator). Kelly: Contractive neural networks Introduction. January 1988 page 2 In 1976 Holden [9] observed that the primary difficulty in the way of attempts to model real neural networks was the lack of data with which to compare models: "The results of these approaches steady states, limit cycles and propagating waves seem trivial compared to Sherrington's idea of 'meaningful patterns', and since there is so little relevant experimental evidence these is a danger that further development of these approaches might become ... less and less relevant to neurophysiology. In the problem of obtaining the behaviour of a net the more playsible the model, the less tractable the system of equations ...." Recent experimental techniques, however, most especially autoradiographic labeling (see [9] and [12]), have made possible observations relating to the electrical activity patterns thought to carry information in neocortex. In particular, the appearance in cortical functional activity patterns of spatial frequencies not present in anatomical observations has been noted [13]. In this paper we consider a relatively simple, though general, class of anatomically-based nonlinear neural network models, containing excitatory and inhibitory synaptic connections of arbitrary reach whose strengths do not change with time, and receiving input of arbitrary form from outside the system. We show that under the assumption of contractiveness (see Section 5 below) such systems respond to input by quickly reaching a steady state which uniquely encodes the input and is independent of the initial state. We show also that under the further assumption that the network is spatially homogeneous, the encoding of the input is equivalent to the action of a filter that enhances certain frequencies characteristic of the network. In addition, we demonstrate that it is possible to construct a synaptic weight pattern whose output spatial frequencies will match those observed to be present in the activity pattern produced by a real network (the forepaw area of S-I in cat) in response to repeated stimulation. The general model we consider is essentially like those considered by Cowan and others (see [5] and [14]). It consists of N units which represent aggregates of neurons; for the purposes of comparison to the experimental data we view the units as cortical "minicolumns", which are approximately 35 JJ (one neuron) in diameter and extend radially through the cortex from upper to lower surface. Associated with each unit, say unit i, is a time-varying quantity Xj(t) representing the average electrical activity of the neurons in the unit over a short time interval containing time 1. With each pair (j,k) of units is a synaptic weight W jk which does not change with time; it represents an average Kelly: Contractive neural networks January 1988 page 3 strength of synaptic transmission from neurons in unit k to neurons in unit j. In realizations of the model the units are viewed as arranged on a two-dimensional grid or in a one-dimensional array, and the synaptic strengths may be near zero for units that are distant from each other. Synaptic transmission is nonlinear: activity in unit k has an effect on that in unit j proportional to WjtA( t), where A is a smooth sigmoid function. Input is constant over time but otherwise arbitrary: each unit, say unit j, receives input from outside the system that tends to change its activity by a fixed amount Pi per unit time, and the quantities Pi can form any pattern whatever, with no restrictions as to magnitude. These assumptions result in the system (1.1) of differential equations governing the behavior of the network. Our results are as follows: 1. Under certain assumptions (primarily contractivenessj see Sections 2 and 5) involving the eigenvalues of the matrix W = {Wjd and the derivatives of the functions ft, the activities Xl (t), x2( t), ... , xN(t) converge at an exponential rate to steady-state activities xl> x2 , ... , xN that depend only on the inputs Pj and not on the initial conditions Xj(O). 2. Under the assumption of spatial homogeneity of synaptic weights (that is, that WiA: is the same as Wr • if the direction and distance of unit k from unit j are the same as those of unit s from unit r), the conditions required in 1 above can be easily checked, and the relation of the steady state to the input can be approximately computed, using discrete Fourier analysis. In fact, the network under these assumptions acts as a filter, enhancing certain characteristic spatial frequencies present in the input pattern and suppressing others. 3. Discrete Fourier analysis of 2-deoxyglucose autoradiographic data ([9], [12]) shows spatial frequencies other than those predicted by known anatomical considerations. Appropriate choice of synaptic weights in the model can cause it to produce steady states with comparable spatial frequencies. eKelly: Contractive neural networks January 1988 page 4 Acknowledgement. The author is greatly indebted to Dr. Barry Whitsel and his co-workers for generously providing data for analysis, and also for numerous illuminating and helpful conversations.